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A  NOTE  ON  ORGANIZATIONAL  LEADERSHIP  AS  PROBLEM  SOLVING 

Introduction 
A  Problem-Solving  Model 

Functional  and  systems  approaches  underlie  much  of  the 
research  on  requirements  for  leadership  performance  in  military 
organizations.  One  body  of  this  research  (Mumford,  1986a; 
Mumford,  1986b;  Mumford,  Yarkin-Levin,  Korotkin,  Wallis, 
Marshall-Mies,  1986)  emerged  as  a  model  of  executive, 
organizational  leadership  (Fleishman,  Mumford,  Zaccaro,  Levin, 
Korotkin,  &  Hein,  1991;  Mumford  &  Connelly,  1991;  Mumford, 
Zaccaro,  Harding,  Fleishman,  &  Reiter-Palmon,  1993) .  This  model 
defined  organizational  leadership  as  discretionary  problem 
solving  in  ill-defined  social  settings.  The  model  further 
specified  a  taxonomy  of  the  leader  behaviors  required  for 
effective  problem  solving.  The  taxonomy  consisted  of  four  broad 
dimensions  of  leader  problem-solving  behavior:  information 
search  and  structuring,  information  use  in  problem  solving, 
managing  personnel  resources,  and  managing  material  resources. 
Expert  judgments  were  obtained  to  decompose  the  four 
superordinate  dimensions  into  more  discrete  descriptions  (or 
representations)  of  the  cognitive  abilities,  knowledge, 
temperament  attributes  (motivation  and  personality) ,  and  complex 
problem-solving  skills  involved  in  effective  problem-solving 
behavior. 

Shortly  after  this  model  of  leadership  problem  solving  had 
been  advanced,  its  construct  validity  was  tested  with  a  sample  of 
slightly  over  1800  U.S.  Army  commissioned  officers,  ranging  in 
rank  from  second  lieutenant  to  full  colonel.  All  officers  were 
then  attending  a  basic  or  more  advanced  course  as  part  of  the 
U.S.  Army's  system  of  institutional  education  and  training.  In 
this  test,  the  predictor  measures  represented  the  requirements 
(cognitive  abilities,  temperament  attributes,  etc.)  set  forth  in 
the  problem-solving  model .  The  criteria  generally  consisted  of 
sslf “^sported  indicators  of  career  achievement  or  of  the  quality 
of  written  responses  to  paper-and-pencil,  problem-solving 
exercises . 

As  of  the  present  report,  results  of  the  construct 
validation  have  not  been  fully  reported.  Recent  presentation 
(Management  Research  Institute,  1995)  has  suggested  that  findings 
generally  supported  the  overall  hypothesis  of  the  model.  This 
hypothesis  is  that  the  cognitive  and  temperament  variables  set 
forth  in  the  model  explain  variance  in  the  career  achievement  and 
problem-solving  effectiveness  of  officers.  A  further  expectation 
from  the  model  is  that  problem-solving  skills  become  increasingly 
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important  with  the  increases  in  decision  discretion  across 
organizational  levels.  From  the  available  presentations,  the 
support  for  this  latter  suggestion  was  less  certain. 

Replication  and  Extension  of  Original  Test 


Rationale 


The  original  test  of  construct  validity  suggested  that  the 
problem-solving  model  of  organizational  leadership  and  its 
measurement  have  potential  for  use  in  the  development  of  Army 
leaders.  Those  presentations  also  made  clear  the  need  for 
further  research  on  the  generalizability  of  initial  findings 
before  decisions  are  made  about  use  of  the  model  and  its 
measures . 

While  the  identification  of  leadership  with  problem  solving 
was  not  totally  new,  the  problem-solving  model  of  organizational 
leadership  provided  a  new  construction  of  the  cognitive 
capabilities  and  temperament  attributes  necessary  for  effective 
leadership.  Many  of  the  measures  of  the  hypothesized  constructs 
were  also  new  and  created  for  the  original  test.  As  with  any 
advance  in  theory  or  measurement,  repeated  testing  of  the  model 
was  needed  to  determine  the  reliability  of  the  original  findings. 

In  addition,  it  seemed  important  to  test  the  model  with 
leaders  who  are  actually  involved  in  organizational  problem 
solving.  That  is,  the  original  test  used  students.  As  such,  the 
students  were  not  at  the  time  actually  involved  in  the  solution 
of  organizational  problems  as  much  as  they  were  engaged  in 
learning  how  to  become  good  leaders.  Given  the  original  sample, 
two  general  types  of  criteria  were  used.  One  type  represented 
career  achievements  (e.g.,  self  reports  of  recommendations  for 
command,  receipt  of  awards,  recommendations  for  early  promotion) . 
The  second  represented  performance  on  paper-and-pencil,  problem¬ 
solving  exercises. 

Even  though  useful,  such  criteria  do  not  yield  clear 
interpretations  about  the  contributions  of  individual-difference 
variables  to  performance  across  levels  of  organizational 
leadership.  The  likelihood  of  receiving  certain  achievements, 
for  example,  varies  with  time.  When  used  to  compare  individuals 
across  levels  of  career  advancement,  a  measure  of  achievement 
could  have  reflected  opportunity  as  much  as  the  quality  of  past 
performance.  Differences  on  paper-and-pencil  exercises  are  also 
open  to  competing  interpretations .  As  found  in  other  educational 
settings,  the  use  of  unstructured  written  exercises  as  an 
instructional  method  likely  increases  from  the  introductory  Army 
courses  (e.g..  Officer  Basic  Course)  to  the  more  advanced  courses 
(e.g..  Command  and  General  Staff  College  or  Senior  Staff 
College) .  As  such,  differences  by  leadership  level  could  have 
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represented  the  practice  which  the  respective  courses  had 
provided  on  written,  open-ended  exercises  like  those  used  as 
criteria . 

Obj  ectives 

The  overall  purpose  of  the  research  reported  here  was  to 
provide  additional  evidence  on  the  problem-solving  model  of 
organizational  leadership.  As  such,  the  research  tested  the 
hypothesis  that  the  cognitive  and  temperament  attributes  set 
forth  in  the  model  explain  variance  in  the  career  achievement  and 
the  problem-solving  effectiveness  of  officers.  This  research 
provided  opportunity  for  only  a  partial  replication  of  the 
original  test  due  to  the  research  sample.  That  is,  the  sample 
for  this  research  was  more  junior  in  level  of  leadership  than  was 
the  original  sample.  This  limitation  was  somewhat  balanced  by 
use  of  officers  in  the  chains  of  command  within  a  sample  of  line 
units.  This  sample  permitted  extension  of  the  criterion  set  to 
the  observed  and  expected  quality  of  the  behavior  of  leaders 
within  the  context  of  organizational  operation. 

Research  Approach 

This  research  tested  the  problem-solving  model  on  officers 
filling  leadership  positions  in  the  chains  of  command  of  Army 
battalions.  Although  there  was  some  variation  in  chain-of- 
command  positions,  those  positions  were  typically  organized 
according  to  a  hierarchy  like  the  following,  in  ascending  order; 
platoon  leader,  company  commander,  and  battalion  commander. 
Accordingly,  within  the  sampled  battalions,  the  officers  serving 
as  platoon  leaders,  company  commanders,  and  battalion  commanders 
responded  to  measures  of  the  cognitive  and  temperament  attributes 
and  provided  self  reports  of  their  achievement.  The  leadership 
of  these  officers  was  rated  by  individuals  who  were  superior  to 
and  subordinate  to  them  in  the  battalion  chain  of  command. 

Predictors 


The  selection  of  cognitive  and  temperament  instruments  built 
on  the  original  test.  During  the  selection,  analyses  of  the 
original  data  were  still  ongoing.  To  enable  cumulative  evidence, 
the  original  researchers  provided  advice  as  to  which  measures 
appeared  most  promising  for  replication.  Advice  on  promise  was 
driven  by  several  general  consideration.  One  consideration  was 
the  likelihood  that  an  instrument  actually  differentiated  leaders 
at  the  levels  sampled  for  this  replication.  This  was  assessed  on 
the  basis  of  the  available  findings  from  the  original  research. 

A  second  concern  focused  on  the  instruments  as  a  set  and  the 
extent  to  which  the  set  covered  all  categories  of  variables  in 
model.  Another  consideration  was  the  time  required  for 
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completion  of  the  instruments.  Administration  requirements 
needed  to  fit  reasonably  with  the  time  likely  available  for 
research  participation.  These  considerations  resulted  in 
measurement  of  the  cognitive  and  temperament  variables  presented 
in  Table  1.  Table  1  also  shows  the  overlap  between  the 
predictors  in  the  original  test  and  the  predictors  in  this 
replication . 

Criteria 


Like  the  original  test,  this  replication  used  indicators  of 
achievement  as  criteria.  Two  self-report  measures  were  used. 

One  was  rank.  The  other  consisted  of  the  awards  which  an  officer 
reported  as  having  received  throughout  his/her  career.  At  the 
outset,  it  was  expected  that  these  two  indicators  are  correlated. 

The  model  under  investigation  characterizes  leadership  as 
organizational  problem  solving  and,  thereby,  hypothesizes  that 
problem  solving,  as  measured  by  the  model,  is  related  to 
assessments  of  the  effectiveness  of  leadership  in  an 
organizational  setting.  This  replication  used  two  measures  of 
leadership  performance  as  criteria.  One  consisted  of  ratings  by 
subordinates  of  an  officer's  transformational  leadership  style, 
as  measured  by  the  Multifactor  Leadership  Questionnaire  (Bass  & 
Avolio,  1991) .  Bass  (1985)  argued  that  compared  to  transactional 
behaviors,  transformational  leadership  enhances  the  performance 
of  organizational  members.  Bass  and  his  colleagues  (e.g.,  Lau, 
Atwater,  Avolio,  &  Bass,  1993)  have  also  demonstrated  that 
transformational  leadership  is  associated  with  leadership  and 
managerial  effectiveness.  A  second  measure  was  a  variant  of 
supervisor  ratings.  That  is,  an  officer's  immediate  superior 
rated  the  target  officer's  "competence"  to  engage  in  types  of 
missions  which,  according  to  military  doctrine  (Department  of  the 
Army,  1993),  units  could  be  called  upon  to  perform.  This 
replication,  thus,  tested  the  expectations  that  the  components  of 
problem-solving  behavior  support  (and  are  positively  associated 
with)  the  performance  of  an  effective  leadership  style  and 
assessments  of  competence  to  perform  organizational  missions. 

Analysis 

Recent  presentation  of  the  original  test  (Management 
Research  Institute,  1995)  described  the  psychometric  properties 
of  the  measures  and  compared  officers  both  within  and  between 
ranks.  Regression  analyses  identified  a  model  of  the  cognitive 
and  temperament  variables  which  parsimoniously  and  best  accounted 
for  variance  in  career  achievement.  The  original  data-analysis 
methods  were  followed  in  this  replication,  to  include  regression 
analyses  to  validate  the  best— fit  model  derived  in  the  original 
presentation. 
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Table  1. 

Crosswalk  of  original  and  replication  measures/variables 


COGNITIVE  ABILITIES 
Standard  Measures 
Verbal  reasoning 
Creative  thinking  capacities 
Writing  skill 
Creative  writing 
New  Measures 

Oral  expression 
Reading  orientation 
Problem  sensitivity 

MOTIVATION/VALUES 
Standard  Measures 
Achievement 
Dominance 
Responsibility 
New  Measures 

Social  recognition 
Social  commitment 
Need  for  affiliation 
Social  alienation 
Need  for  approval 


PERSONALITY 
Standard  Measures 
Extroversion 
Sensing/intuiting 
Thinking/feeling 
Judging/perceiving 
Openness 
Ego  resiliency 
New  Measures 

Self  discipline 

Rigidi ty 

Locus  of  control 

Ego  resiliency 

Competitiveness 

Environmental  engagement 


KNOWLEDGE 
New  Measures 

Leadership  expertise 
Leadership  problem  solving 

SOCIAL  JUDGMENT  SKILLS 
New  Measures 

Social  judgment 
Self  reflectivity 
Insightfulness 
Good  judgment 
Systems  perceptiveness 

PROBLEM-SOLVING  SKILLS 
New  Measures 

Problem- const ruction 
Problem-solving  component  skills 
Creative  problem  solving 

PRACTICAL  INTELLIGENCE 
New  Measures 

Troubleshooting 
Planning  under  ambiguity 
Monitoring 

Information  gathering 
Selection  of  solution  components 

SOCIAL  INTELLIGENCE 
New  Measures 

Interpersonal  perceptiveness 
Social  adroitness 
Harmony  facilitation 
Behavioral  flexibility 


Note! This  table  lists  the  measures  administered  in  the  original  project. 
Only  those  measures  typed  in  italics  were  administered  in  this  replication. 
In  neither  the  original  nor  the  replication  research  were  standard  measures 
administer  for  the  following  categories:  knowledge,  social  judgment  skills, 
problem  solving  skills,  practical  intelligence,  and  social  intelligence. 
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Method 


Participants 

Data  were  collected  on  leaders  in  the  chains  of  command  of 
53  battalions  across  eight  Army  posts.  The  research  plan  called 
for  administration  of  the  problem-solving  predictors  to  the 
following  per  battalion:  the  battalion  commander  (BNC) ,  4  company 
commanders  (CCs),  and  16  platoon  leaders  (PLs) .  The  planned 
sample  also  included  superiors  and  subordinates  of  the  chain-of- 
command  officers:  the  brigade  commander  (BGDC)  of  each  BNC,  4 
battalion  staff  officers  (BSs)  under  the  BNC,  and  10  platoon 
members  serving  under  each  PL  sampled. 

The  obtained  sample  of  chain-of-command  leaders  consisted  of 
53  BNCs,  227  CCs,  and  490  lieutenants  in  PL  or  other  entry 
positions.  A  total  of  3843  platoon  members  across  469  platoons 
and  144  BSs  across  47  battalions  rated  the  transformational 
leadership  of  their  superiors  (PLs  and  BNCs,  respectively) . 

BGDCs  provided  superior  ratings  of  35  BNCs. 

Table  2  describes  the  returns  for  the  measures  targeted  on 
the  chain-of-command  leaders.  Officers  reported  demographic  data 

Table  2 . 

Distribution  of  Obtained  Chain-of-Coininand  Leaders  by  Rank  and  Type  of  Measure 


Predictors  Criteria 


Rank 

Back¬ 

ground 

Data 

Demo¬ 

graphics 

Form 

Problem- 

Solving 

Exercises 

Superior 

Ratings 

Subordinate 

Ratings 

01/02 

312 

322 

319 

409 

333 

03 

162 

166 

166 

196 

192 

05 

41 

41 

36 

35 

50 

Note.  The  numbers  of  01/02s,  03s,  and  05s  with  data  on  all  types  of  measures 
(listwise  complete)  were,  respectively,  169,  199,  and  22. 

on  a  separate  demographic  form  or  on  the  form  used  to  rate  their 
superior  leader .  Of  the  607  chain-of-command  officers  reporting 
gender,  only  13  were  female.  Mean  tenure  in  the  Army  ranged  from 
25.7  months  for  second  lieutenants  to  224.5  months  for  BNCs. 
Tenure  in  present  battalion  was  7.7  months  for  second 
lieutenants,  14.4  months  for  first  lieutenants,  11.8  months  for 
captains,  and  11.9  months  for  lieutenant  colonels.  The 
predominate  commissioning  source  was  the  Reserved  Officer 
Training  Corps  (n  =  342),  followed  by  the  U.S.  Military  Academy 
(n  =  139)  and  Officer  Candidate  School  (n  =  52) . 
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Measures 


Predictors 


The  chain-of-command  leaders  provided  demographic  data  and 
responded  to  instruments  that  measured  the  cognitive  and 
temperament  variables  set  forth  as  predictors  of  leader  problem¬ 
solving  effectiveness.  One  instrument  measured  general  cognitive 
abilities.^  Three  instruments  measured  complex  problem-solving 
skills.  The  third  instrument  contained  background-data  measures 
which  were  developed  for  the  original  investigation  and  which 
were  intended  to  measure  personality,  motivational,  and  a  variety 
of  cognitive  variables. 

Demographic  description  The  chain-of-command  officers 
completed  a  demographic  form  which  provided  spaces  for  entering 
their  present  officer  grade  (or  rank) ,  present  unit  and  position, 
number  of  months  on  active  duty  in  the  Army,  number  of  months  in 
their  current  position,  and  other  background  data.  This  form 
also  contained  the  items,  described  later,  used  to  count  the 
awards  received  during  their  career. 

Problem-solving  predictors  General  cognitive  ability  was 
assessed  by  a  12-item  version  of  a  standardized  measure  of  verbal 
reasoning,  the  Employee  Aptitude  Survey  (EAS)  Verbal  Reasoning 
Test.  This  version  was  reported  in  the  original  validation  as 
having  reliability  in  the  .70s  (cf.  Mumford  et  al . ,  1993).  As  in 
the  original  validation,  an  individual's  responses  were  scored  by 
counting  the  number  of  correct  responses  and  by  translating  the 
obtained  raw  score  into  a  score  on  a  standard  distribution.  The 
transformation  produced  scores  ranging  from  0  to  45. 

Responses  to  open-ended  problem-solving  exercises  were  used 
to  assess  three  sets  of  complex  problem-solving  skills:  problem 
construction,  social  judgment,  and  component  problem-solving 
skills.  Each  exercise,  one  for  each  set  of  skills,  consisted  of 
one  or  two  scenarios .  Each  exercise  required  respondents  to  read 
a  scenario  and  then  give  written  responses  to  questions  posed 
about  the  situation  described  in  the  scenario.  The  scenarios  and 
questions  in  each  exercise  had  been  constructed  for  the  original 
investigation.  An  exercise  was  intended  to  elicit  responses 
which  could  be  scored  for  dimensions  or  elements  of  the  complex 
skills  targeted  by  the  exercise.  Exercise  completion  was  timed 


1  Both  the  original  construct  validation  and  this  replication 
included  a  second  measure  of  general  ability.  This  was  a 
standardized  measure  of  spatial  ability.  Results  on  spatial 
ability  were  not  reported  in  the  most  recent  presentation  of  the 
original  validation.  Following  the  original  format,  the  present 
report  also  excluded  results  on  spatial  ability. 
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so  that  officers  were  allowed  10  minutes  to  respond  to  the  one  or 
two  scenarios  in  an  exercise. 

Responses  to  the  exercises  were  rated  per  scenario.  Table  3 
lists  the  variables,  by  problem-solving  skill,  used  to  rate  the 
scenarios.  The  variables  in  Table  3  are  the  dimensions  (or 
elements)  which,  by  the  model  under  investigation,  formed  or 
otherwise  characterized  the  skills.  Rating  involved  reading  the 
responses  given  to  a  scenario  and  (depending  on  the  exercise  and 
the  variable)  using  a  5-point  or  6-point  scale  to  assign  a  value 


Table  3 

Dimensions  Used  to  Rate  Problem-Solving  Exercises 


Exercise 

Cognitive  Skills 

Dimensions 

Measured 

Rated 

Military 

Problem  Construction 

-Short  vs  long 

Scenarios 

term  implications 
-Attention  to 
restrictions 
-Self  oriented  goals 
-Organizationally 
oriented  goals 
-Quality 
-Objectivity 
-Number  of 
alternatives 
-Originality 

Organizational 

Social  Judgment 

-Self  objectivity 

Scenarios 

Skills 

-Self  reflectivity 
-Sensitivity  to  fit 
-Systems  perspective 
-Good  judgment  under 
conditions  of 
uncertainty 
-Systems  commitment 
-Overall  wisdom 

Problem 

Problem-Solving 

-Problem  construction 

Solving 

Component  Skills 

-Information  encoding 

Processes 

-Category  search 
-Specification  of 
best  fitting 
categories 
-Combination  and 
reorganization 
-Idea  evaluation 
-Implementation 
-Monitoring 
-Overall  quality 
“Overall  originality 
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indicating  the  extent  to  which  (or  how  well)  the  dimension  was 
represented  in  the  responses. 

The  open-ended  responses  were  rated  by  individuals  who  were 
trained  by  others  having  served  as  raters  in  the  original 
investigation.  Training  involved  discussion  of  the  variables 
used  to  rate  the  scenarios,  practice  in  making  ratings,  and 
further  discussion  of  similarities  and  differences  in  the  ratings 
made.  The  overall  aim  of  the  training  was  to  develop  common 
meaning  and  application  of  dimensions  in  terms  of  the  scenarios 
contained  in  the  exercises. 

A  total  of  six  different  individuals  rated  responses  to  the 
exercises.  Responses  were  assigned  so  that  each  officer's 
response  to  each  exercise  was  evaluated  independently  by  three 
trained  raters.  Table  4  describes  the  raters  who  scored  some 
proportion  of  the  responses  to  the  three  exercises.  As  Table  4 
shows,  four  individuals  served  as  raters  for  two  exercises.  The 
other  two  individuals  scored  some  proportion  of  the  responses  to 
either  only  one  exercise  or  to  all  three. 


Table  4 . 

Distribution  of  Raters  by  Problem-Solving  Exercise 


Exercise 


Problem 

Social  Judg- 

Problem 

Rater 

Construction 

ment  Skills 

Solving  Process 

A 

+ 

+ 

+ 

B 

+ 

- 

+ 

C 

- 

+ 

+ 

D 

+ 

+ 

- 

E 

+ 

+ 

F 

+ 

Note .  Entries  indicate  whether  an  individual  rater  did  (+)  or 
did  not  (“)  score  an  exercise. 
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Background  data  inventory.  The  background  data  inventory 
(BDI)  contained  questions  about  certain  life  events  and 
experiences  and  about  one's  behavior  and  feeling  in  past 
situations.  Respondents  answered  by  choosing,  from  the 
alternatives  given,  the  particular  alternative  that  best 
described  the  extent  to  which  the  experience  had  typically 
applied  to  them'  personally.  The  BDI  had  been  constructed  for  the 
original  investigation  to  measure  variables  in  the  domains  of 
personality,  motivation,  and  cognition  (to  include  general 
cognitive  skills,  practical  intelligence,  social  intelligence, 
and  social  judgment  skills) . 

A  reduced  version  of  the  original  BDI  was  used  in  this 
replication.  The  reduction  was  based  on  results  of  factor 
analyses  of  the  BDI  data  collected  in  the  original  investigation. 
More  specifically,  for  each  domain  which  the  BDI  had  been 
designed  to  measure,  a  separate  factor  analysis  was  conducted  on 
responses  to  the  items  which  were  expected  to  measure  the 
variables  in  the  domain.  For  example,  responses  to  the  items 
expected  to  measure  the  four  variables  in  the  domain  of  social 
intelligence  (interpersonal  perceptiveness,  social  adroitness, 
harmony  facilitation,  and  behavioral  flexibility)  were  included 
in  a  separate  factor  analysis.  The  175  items  with  strongest 
loadings  on  the  expected  variables  (in  general,  the  top  6  items 
with  eigen  values  of  .30  or  greater  for  each  factor  obtained) 
formed  the  BDI  for  this  replication. 

Criteria 


Career  Achievement.  Two  indicators  of  career  achievement 
served  as  criteria:  self-reported  rank  and  awards.  Officers 
reported  their  grade  on  the  demographic  form.  On  this  same  form, 
officers  also  checked  whether  they  had  ever  during  their  career 
received  each  of  the  medals  on  the  following  list:  Army  Good 
Conduct  Medal,  Meritorious  Service  Medal,  Defense  Meritorious 
Service  Medal,  Army  Commendation  Medal,  Letter  of  Commendation, 
Army  Achievement  Medal,  Silver  Star,  Bronze  Star,  Air  Medal, 
Purple  Hear,  Medal  of  Honor,  Legion  of  Merit,  Campaign  Medal, 
Other.  This  list  was  among  the  set  of  verifiable  indicators  of 
leader  achievement  developed  for  the  original  validation.  For 
the  present  research,  awards  received  was  measured  as  the  total 
count  of  medals  checked  on  this  list. 

Leadership  Performance.  Leadership  performance  was 
evaluated  by  both  a  leader's  superior  officer  and  his  or  her 
subordinates.  For  superior  evaluations,  chain-of-command  leaders 
assessed  the  "competence"  of  their  subordinate  leaders  "to  engage 
in"  types  of  military  missions.  The  rating  form  contained  a 
table  of  missions  by  target  leader  rated.  The  eight  missions, 
presented  as  the  table  rows,  were  based  on  missions  that  the  Army 
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could  be  called  upon  to  perform,  as  described  in  the  U.S.  Army 
Field  Manual  100-5  Operations  (1993) .  These  mission  types  were: 
mid  intensity  attack,  mid  intensity  defense,  low  intensity 
attack,  low  intensity  defense,  noncombatant  evacuation, 
humanitarian  assistance  &  disaster  relief,  support  to  counter 
drug  operations,  and  combating  terrorism.  The  columns  of  the 
table  contained  spaces  in  which  a  leader  entered  the  subordinates 
to  be  evaluated.  CCs  entered  designations  for  and  evaluated 
their  PLs .  BNCs  designated  and  evaluated  CCs  and  PLs  in  their 
commands.  BNCs  used  one  form  to  evaluate  all  CCs  and  separate 
forms  to  rate  PLs  in  each  company.  BGDCs  evaluated  their 
subordinate  BCs.  To  complete  the  table,  superior  officers 
assessed  each  subordinate's  competence  for  performing  each 
mission  on  a  five-level  scale  anchored  (and  scored)  as:  fair 
(1) ,  good  (2),  very  good  (3),  excellent  (4),  and  best  of  all  (5). 
Leaders'  were  instructed  to  use  the  following  as  a  comparison  in 
assessing  a  subordinate  leader:  "all  other  officers  you  have 
known  of  about  the  same  age,  rank,  and  command  position".  A 
superior's  evaluation  of  a  subordinate  leader  was  computed  as  the 
average  of  the  superior's  assessments  across  missions  for  the 
subordinate . 

Subordinate  ratings  on  the  Multifactor  Leadership 
Questionnaire  (Bass  &  Avolio,  1991)  were  used  to  assess  the 
transformational  leadership  behaviors  of  their  immediately 
superior  officer.  The  forty-seven  items  for  the  five  dimensions 
of  transformational  leadership  were  averaged  to  form  a  single 
scale.  Scale  reliability  was  high  for  each  type  of  subordinate 
rater  (a  coefficient  a  of  .97  or  greater).  The  ratings  of  like 
subordinates  (for  example,  platoon  members)  of  a  particular 
leader  (for  example,  PL)  were  aggregated  to  form  a  single  score 
for  each  target  leader  and  for  each  type  of  subordinate  rater. 

Properties  of  Criteria.  Table  5  shows  correlations  among 
the  four  criteria.  Rank  and  awards  were  strongly  correlated. 

Rank  and  awards  were  also  positively  and  significantly  correlated 
with  the  leadership  performance  assessments,  with  this 
relationship  moderately  strong  for  subordinate  assessments.  The 
correlation  between  superior  and  subordinate  ratings  of 
leadership  performance  was  significant  but  weak. 

Correlations  between  raters  were  computed  for  leaders 
assessed  by  more  than  one  rating  group.  The  evaluations  of  PL 
leadership  by  BNCs  and  CCs  were  weakly  correlated  (r  =.26,  p  < 
.05,  n  =  332),  and  partialing  PLs'  time  in  the  battalion  somewhat 
strengthened  this  relationship  (r  =  .35,  p  <  .05,  n  =  203).  BSs' 
and  CCs'  assessments  of  BNCs'  transformational  leadership  yielded 
moderately  strong  correlations  (r  =  .41,  p  <  .05,  n  =  44),  and 
this  moderately  strong  relationship  was  not  changed  substantially 
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after  having  controlled  for  the  EC's  time  in  the  battalion  (r  = 
. 43,  £  <  . 05,  n  =  28)  .  “ 


Table  5. 

Means,  Standard  Deviations,  and  Correlations  between  Criteria 


Standard  Correlations 


Mean 

Deviation 

1 

2 

3 

1) 

Rank 

2.20 

1.20 

2) 

Awards 

2.25 

1.87 

.70* 

■  — 

3) 

Subordinate 

Evaluations 

3.28 

.63 

.50* 

* 

.41 

— 

4) 

Superior 

Ratings 

3.20 

.88 

.23* 

* 

.13 

* 

.11 

*  £  <  .05. 


Procedures 


About  one  month  prior  to  on-site  data  collection  at  a  post, 
envelopes  containing  demographic  information  forms  and  background 
data  surveys  were  sent  for  distribution  to  each  chain-of-command 
leader  designated  for  research  participation.  The  packets 
included  instructions  explaining  the  purpose  of  the  research. 

The  packets  also  instructed  officers  to  complete  the  enclosed 
materials  at  their  convenience  and  to  take  the  completed 
materials  to  an  in-class  session  scheduled  for  their  battalion. 

A  two-hour,  in-class  session  was  scheduled  for  the  chain-of- 
command  leaders  in  each  battalion.  In  the  in-class  session,  the 
officers  completed  the  two  cognitive  measures  and  the  three  open- 
ended  problem-solving  exercises .  The  CCs  and  PLs  then  responded 
to  the  MLQ  to  rate  the  leadership  behavior  of  their  immediate 
superior  in  the  chain-of-command.  While  CCs  and  PLs  responded  to 
the  MLQ,  the  BNC  provided  superior  evaluations  of  the  leadership 
of  the  CCs  and  PLs  within  their  battalion.  After  responding  to 
the  MLQ,  each  CC  completed  superior  evaluations  of  the  leadership 
of  each  PL  in  their  company. 

A  separate  in-class  session  was  held  for  the  PMs  and  BSs  in 
each  battalion.  After  instructions  on  the  purpose  of  the 
research,  PMs  and  BSs  responded  to  the  MLQ  to  rate,  respectively, 
the  leadership  behavior  of  their  PL  and  BNC. 
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Office  appointments  were  scheduled  with  BGDCs .  During  the. 
office  calls,  the  purposes  of  the  research  and  of  the  superior 
evaluations  were  explained.  BGDCs  also  received  the  form  for 
evaluating  the  leadership  of  the  subordinate  BNCs  with  battalions 
included  in  the  research.  Depending  on  a  BGDCs  availability, 
the  BGDC  either  completed  the  form  during  the  office  call  or  kept 
the  form  for  later  completion  and  return. 

Results 

Properties  of  the  Measures 

Table  6  groups  the  predictor  variables  by  type  of  measure 
(background  data  inventory  (BDI)  or  problem-solving  exercise)  and 
displays  the  obtained  descriptive  statistics. 

BDI  Scales 


As  described  earlier,  the  BDI  scales  used  in  this 
replication  contained  a  smaller  number  of  items  than  the  version 
used  originally.  Factor  analyses  (principal  components  analyses 
with  oblimin  rotations)  were  conducted  to  examine  the  stability 
of  the  expected  BDI  scales.  A  separate  factor  analysis  was 
conducted  for  responses  to  the  retained  items  in  each  domain 
(e.g.,  the  retained  motivation  items) .  One  exception  was  that 
the  cognitive  and  practical  intelligence  items  were  factor 
analyzed  together  because  of  the  apparent  similarity  of  the  items 
in  these  two  dimensions;  the  sample  size  also  permitted  a 
combined  analysis  (i.e.,  a  greater  than  10  to  1  ratio).  From 
these  analyses,  scales  were  formed  to  represent  the  obtained 
factors.  Scales  were  formed  by  averaging  the  responses  to  items 
which  had  rotated  factor  loadings  of  .30  or  greater  and  which  had 
been  used  in  the  original  research  to  form  a  scale  (or  in  the 
case  of  the  BDI  scale  of  "problem  solving",  the  items  which 
formed  the  original  scales  captured  by  the  problem-solving  factor 
obtained  in  this  replication) . 

Table  6  describes  the  number  of  items  and  reliability  of 
each  scale.  The  replicated  scales  and  the  lengths  of  those 
scales  differed  from  some  of  the  original  scales  due  to:  a) 
differences  in  the  results  of  the  factor  analysis  and  b) 
adjustments  by  the  original  researchers  to  the  procedures  for 
combining  items  prior  to  their  more  recent  presentation  but  after 
items  had  been  selected  for  this  replication.  The  reliabilities 
of  the  obtained  scales  compared  favorably  with  the  reliabilities 
reported  for  the  original  BDI  scales. 

Table  7  shows  correlations  between  the  BDI  scale  scores 
obtained  in  this  replication.  It  is  not  clear  that  given  the 
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Table  6. 

Descriptive  Statistics  for  Measures  of  Predictor  Variables 
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Table  7 . 

Correlations  between  BDI  Scales 
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Note.  With  an  approximate  sample  size  of  515,  all  correlations  of  +.09  were  statistically  significant  (p  <  .0! 
Also  statistically  significant  (p  <  .05)  were  the  two  .08  correlations  between  reading  orientation  and  harmony 
facilitation  and  between  social  recognition  and  behavioral  flexibility  were  . 


domains  covered  by  the  BDI,  one  would  necessarily  expect  that  the 
correlations  between  the  variables  within  the  same  domain  would 
be  stronger  than  the  correlations  between  the  variables  in 
different  domains.  To  the  extent  such  an  expectation  makes 
sense,  however,  the  correlations  in  Table  7  tend  to  question  the 
divergence  of  the  measures  yielded  by  the  BDI,  if  not  the  model 
underlying  it.  For  example,  the  BDI  was  constructed  to  measure 
components  of  both  social  intelligence  and  social  judgment.  If 
these  are  indeed  separate  constructs,  it  seems  reasonable  to 
expect  a  pattern  of  relatively  stronger  correlations  between  the 
components  within  a  construct  than  across  constructs.  Such  a 
pattern  was  not  clearly  obtained.  Examination  revealed  that  the 
range  of  the  absolute  values  of  the  correlations  between 
components  of  social  intelligence  was  .07  -  .44,  with  a  median  of 
about  .26.  For  social  judgment,  the  range  and  median  for 
absolute  values  were  .01  -  .33  and  .08,  respectively.  These 
within-domain  statistics  were  hardly  distinguishable  from  the 
comparable  statistics  for  the  absolute  values  of  the  across- 
domain  correlations.  The  across-domain  range  was  .03  -  .50,  and 
the  cross-domain  median  was  .20. 

Measures  Derived  from  Problem-Solving  Exercises 

The  final  score  for  each  of  three  problem-solving  exercises 
was  derived  according  to  the  procedures  in  the  most  recent 
presentation  of  the  original  research.  For  each  of  the 
exercises,  this  score  was  computed  by  averaging  (across 
scenarios)  the  scores  assigned  to  each  dimension  by  the  three 
raters  for  a  exercise.  These  dimension  averages  were  then  again 
averaged  to  yield  a  total  score  for  the  exercise.  It  should  be 
noted  that  raters  had  been  instructed  not  to  score  responses  to  a 
scenario  (or  to  a  particular  question)  when  the  responses  were 
too  incomplete  to  make  judgments  on  the  scoring  dimensions.  Due 
to  these  instructions,  scores  for  some  respondents  were  based  on 
ratings  for  a  reduced  number  of  dimensions,  and  the  scores  were 
not  adjusted  for  the  completeness  of  a  response. 

The  reliability  of  ratings  of  the  open-ended  problems- 
solving  exercises  was  assessed  by  the  procedures  used  in  the  more 
recent  presentation  of  the  original  research:  intarclass 
correlations  (Shrout  &  Fleiss,  1979)  of  the  total  exercise 
scores.  By  these  procedures,  reliability  was  assessed  at  the 
level  of  the  overall  construct  measured  by  an  exercise  and  at  the 
level  of  the  aggregation  which  produced  the  overall  measure. 

The  intarclass  correlations  obtained  for  the  three  open-ended 
measures  ranged  from  .71  to  .81.  This  range  was  comparable  to 
that  reported  in  the  original  research  (i.e.,  .67  to  .82). 
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Table  8  presents  correlation  between  the  three  scores 
derived  from  the  open-ended  exercises.  Table  8  also  includes  the 
correlations  with  the  other  cognitive  predictors,  verbal 
reasoning  and  the  BDI  scale  of  reading  orientation.  Table  8 
shows  that  the  significant  correlations  between  the  three  open- 
ended  measures  were  positive  but  moderately  strong  at  best. 


Table  8. 

Correlations  between  Cognitive  Measures 


Measure 

1 

2 

3 

4 

5 

1)  Problem  construction 

— 

.36* 

.30* 

.07 

.05 

2  Social  judgment 

— 

.34* 

.13* 

.04 

3)  Component  skills 

— 

.19* 

.15 

4)  Verbal  reasoning 

— 

.18 

5)  Reading  orientation 


*  £  <  .05. 


Two  of  the  open-ended  exercises  were  significantly  correlated 
with  verbal  reasoning  were  not  statistically  significant.  The 
BDI  "reading  orientation"  scale  was  positively  and  significantly 
correlated  with  only  one  of  the  three  open-ended  measures 
(component  problem-solving  skills)  and  with  verbal  reasoning.  The 
correlations  with  verbal  reasoning  and  reading  orientation  were 
no  greater  than  .19  and,  thus,  relatively  weak. 

The  data  in  Table  8  suggested  greater  consistency  within 
measurement  method.  As  a  further  examination,  each  dimension 
rating  for  each  of  the  five  scenarios  in  the  open-ended  exercises 
(two  scenarios  for  problem  construction,  two  for  social  judgment, 
and  one  for  component  problem-solving  skills)  was  treated  as  a 
score.  This  score  was  computed  (per  respondent)  as  the  average 
of  the  ratings  which  the  three  raters  had  assigned  to  the 
dimension  and  for  the  scenario.  These  scores  were  then  entered 
into  a  factor  analysis  (principal  components,  oblimin  rotation) . 
Six  factors  emerged,  with  the  factors  tending  to  represent  the 
scenario  to  which  the  rating  had  been  assigned.  More 
specifically,  one  factor  emerged  for  each  of  the  four  scenarios 
comprising  the  social  judgment  and  the  problem  construction 
exercises.  Two  additional  factors  reflected  the  first  and  second 
half  of  the  measure  of  component  problem-solving  skills. 
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Results  of  the  factor  analysis  are  perhaps  reflected  in 
Table  9  which  contains  correlations  between  the  scores  on  each 
scenario.  Based  on  these  correlations,  it  appears  that  the 
scores  for  scenarios  in  the  same  exercise  were  relatively  more 
strongly  correlated  than  were  the  scores  for  scenarios  in 
different  exercises.  However,  the  correlations  within  exercises 
were  at  best  moderately  strong  (range  of  .41  to  .43).  Together 


Table  9. 

Correlations  between  Ratings  of  Separate  Scenarios  in  Problem-Solving 
Exercises 


Exercise 

1 

2 

3 

4 

5 

1)  Problem  Construction 
Scenario  1 

— 

.43 

.19 

.26 

.23 

2)  Problem  Construction 
Scenario  2 

— 

,24 

.31 

.26 

3)  Social  Judgment 

Scenario  1 

— 

.41 

.25 

4)  Social  Judgment 

— 

.31 

Scenario  2 

5)  Component  Skills 
(One  Scenario) 


Note.  All  correlations  statistically  significant  (p  <  .05).  Sample  per 
correlation  varied  from  537  to  584  leaders. 


with  results  of  the  factor  analyses,  these  correlations  tend  to 
question  the  measures,  that  is,  whether  the  constructs  intended 
for  measurement  or  some  other  property  of  the  exercises  drove  the 
open-ended  responses. 


Relationships  between  BDI  and  Open-Ended  Predictors 

Within  the  validation  framework,  the  BDI  scales  representing 
cognitive  ability,  practical  intelligence,  and  social 
intelligence  were  expected  to  predict  officers'  social  problem¬ 
solving  skills  and  career  Army  achievement.  As  such,  it  is 
appropriate  to  examine  correlations  between  the  BDI  scales  and 
the  cognitive  and  problem-solving  skills  measured  by  the  open- 
ended  exercises.  As  contained  in  Table  A-1,  Appendix  A,  the  BDI 
scale  of  "problem  solving"  was  significantly  correlated  with  two 
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of  the  problem-solving  exercises.  However,  these  correlations 
were  relatively  weak  (at  best,  .20),  and  consistent  relationships 
were  not  obtained  for  any  of  the  other  BDI  scales  related  to 
cognitive  ability,  intelligence  (social  or  practical),  and  social 
j  udgment . 


Test  of  the  Problem-Solving  Model/Measures 

Two  sets  of  analyses  were  conducted  to  test  relationships 
among  predictors  and  criteria.  First  the  raw  correlations 
between  the  predictor  and  criterion  sets  were  computed  and 
compared  with  those  reported  originally.  Second,  regression 
analyses  were  conducted  to  test  the  significance  of  the 
parsimonious  model,  derived  in  the  original  research,  with  the 
expanded  criteria  collected  as  part  of  this  replication. 

Correlations  between  Predictors  and  Criteria 


Table  10  presents  the  correlations  between  predictors  and 
criteria  obtained  in  this  replication.  Table  10  includes  the  22 
measures  which  were  common  to  the  two  data  collections.  It  also 
contains  correlations  for  the  BDI  scale  of  problem  solving 
derived  in  this  replication.  The  problem-solving  scale  was 
computed  from  items  which,  in  the  original  effort,  had  been  used 
to  form  five  separate  scales. 

Similarities  between  the  original  and  replication 
correlations  were  evident  for  the  measures  of  career  achievement. 
Of  the  22  common  predictors,  21  were  significantly  associated 
with  career  achievement  in  the  original  data  collection.  Of 
those  21  predictors,  12  in  this  replication  were  significantly 
correlated  with  rank,  self-reported  awards,  or  both.  The  BDI 
scale  of  problem  solving  was  significantly  correlated  with  both 
rank  and  self-reported  rewards;  the  original  effort  found  that 
three  of  the  scales  with  items  contributing  to  problem  solving 
were  significantly  correlated  with  the  career  achievement. 

Despite  the  correspondence  of  results  for  career  achievement 
criteria,  the  correlations  obtained  in  this  replication  tended  to 
be  weaker  than  those  obtained  originally.  This  attenuation  was 
especially  noticeable  for  the  measures  of  problem-solving  skills 
derived  from  the  open-ended  exercises. 

In  sharp  contrast  to  the  correlations  with  rank  and  awards, 
few  predictors  were  significantly  associated  with  superior  and 
subordinate  assessments  of  leadership  performance.  Of  the  46 
correlations,  only  five  were  statistically  significant  and  in  the 
expected  direction  (social  alienation,  planning  under  ambiguity, 
problem  solving,  systems  perceptiveness,  and  problem 
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Table  10. 

Correlations  between  Predictors  and  Criteria 
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correlations  of  .09  or  greater  were  statistically  significant  (£  <  .05).  For  other 
ons  of  .10  or  greater  reached  significance. 


construction) .  Another  two  correlations  (social  commitment  and 
self  reflectivity)  were  statistically  significant  but  not  in  the 
expected  direction. 

Correlations  of  predictors  with  leadership  ratings  within 
levels  of  the  chain  of  command  are  presented  in  Table  A-2, 
Appendix  A.  Because  officers  at  the  same  level  were  of  similar 
rank,  these  analyses  provided  a  control  for  the  effects  of  rank. 
Of  the  184  correlations  in  Table  A-2,  only  16  were  statistically 
significant,  and  no  consistent  pattern  was  discernible. 

Regression  Analyses 

In  the  original  effort,  a  series  of  regression  analyses  were 
conducted  to  identify  the  predictor  sets  which  reduced  the  number 
of  possible  predictors  while  maximally  accounting  for  career 
achievement.  Due  to  variations  in  the  open-ended  problem-solving 
exercises  completed  by  officers  in  the  original  effort,  two 
general  sets  of  models  were  actually  derived.  One  set  applied  to 
the  portion  of  the  original  sample  which  had  taken  the  exercise 
measuring  "problem  construction".  The  other  set  was  based  on  the 
respondents  who  had  completed  the  exercise  measuring  "social 
judgment".  Simultaneous  regression  analysis  showed  that  the 
models  accounted  for  up  to  36%  of  the  variance  in  career 
achievement . 

The  original  models  with  the  measure  of  "problem 
construction"  were  chosen  for  repeated  testing.  These  models 
were  chosen  because  their  predictors  mapped  more  readily  onto  the 
measures  collected  in  this  replication.  Their  use  also  reduced 
the  risks  of  prematurely  rejecting  the  original  findings  in  that 
the  original  models  with  problem  construction  had  been  relatively 
stronger  than  the  original  model  with  social  judgment. 

Minimizing  those  risks  seemed  justified  given  the  attenuated 
correlations  found  in  this  replication. 

To  test  the  models,  the  variables  which  most  closely 
approximated  those  in  the  original  models  were  identified.  This 
resulted  in  selection  of  the  following  variables  for  inclusion  in 
the  model:  the  measures  of  problem-solving  component  processes 
and  problem-construction  based  on  the  open-ended  exercises  and 
the  BDI  scales  measuring  planning  under  ambiguity,  reading 
orientation,  rigidity,  interpersonal  perceptiveness,  and  problem¬ 
solving.  The  BDI  problem-solving  scale  included  items  on  several 
of  the  separate  BDI  scales  included  in  the  original  models. 
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Table  11  summarizes  results  of  the  regression  analyses 
conducted  in  this  replication.  The  predictor  set  significantly 
predicted  all  measures  of  career  achievement  and  leadership 


Table  11. 

Results  of  Predictor  Model 


Predictor 

Achievement 

Rank  Awards 

Leadership 
Performance  Ratings 
Superior  Follower 

BDI  Scales 

Reading  orientation 

-.07 

-.03 

-.03 

-.08 

Rigidity 

-.11* 

-.09* 

.00 

-.05 

Planning  under  ambiguity 

.18* 

.18* 

.07 

.13* 

Interpersonal  perceptiveness 

-.06 

-.03 

-.04 

-.01 

Systems  perceptiveness 

.17* 

.18* 

.17* 

.02 

Problem  solving 

.08 

.06 

.02 

.04 

Problem-Solving  Exercises 

Problem  construction 

.24* 

.26* 

.06 

.09 

Component  skills 

-.02 

-.01 

.06 

.08 

r2 

.14* 

.14* 

•04* 

.04* 

(n) 

(443) 

(417) 

(338) 

(451 

Note.  Except  as  otherwise  indicated,  entries  are  standardized  £  weights, 
*£  <  .05 


performance.  While  statistically  significant,  the  contributed 
variances  was  at  best  14%,  and  the  level  of  contribution  appeared 
to  be  consistently  greater  for  the  criterion  measures  of  career 
achievement  (14%)  than  for  the  measures  of  leadership  performance 
(4%) .  Moreover,  the  obtained  levels  of  prediction  were 
substantially  smaller  than  the  originally  reported  estimates  of 
32%  to  36%  for  the  similar  predictor  sets.  Of  the  184 
correlations  in  Table  A2,  only  16  were  statistically  significant, 
only  seven  more  than  expected  by  chance.  Of  those  16,  five  were 
in  the  unexpected  direction. 

Discussion 

Results  of  this  replication  provided  some  support  for  the 
hypothesis  that  the  cognitive  and  temperament  variables  set  forth 
in  the  problem-solving  model  of  leadership  explain  variance  in 
the  leadership  of  officers.  The  variables  were  significantly 
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correlated  with  the  achievement  measures  of  rank  and  awards,  and 
the  prediction  equations  for  the  achievement  measures  were 
significant.  Despite  these,  all  relationships  with  achievement 
were  smaller  in  magnitude  than  had  been  obtained  originally. 
Moreover,  the  original  findings  were  not  replicated  for  the  more 
direct  measures  of  leadership,  the  assessments  made  by 
subordinates  and  superiors.  The  separate  variables  were  not 
consistently  related  to  subordinate  or  superior  assessments,  and 
relationships  of  the  predictors  as  a  group  to  such  assessments 
were  weak. 

Several  limitations  of  this  field  test  possibly  contributed 
to  the  lack  of  consistent  relationships  with  the  measures  of 
leadership  performance.  First,  the  relatively  junior  status  of 
the  replication  sample  (predominately  lieutenants  and  captains) 
may  have  attenuated  relationships.  That  is,  a  premise  of  the 
model  is  that  problem-solving  requirements  are  somewhat  greater 
for  more  senior  leaders.  Given  this  premise,  the  model  is 
possibly  most  descriptive  of  senior  leaders,  and  stronger 
relationships  would  have  possibly  been  obtained  if  the  sample  had 
consisted  of  more  senior  leaders.  Second,  the  method  for 
measuring  superior  evaluations  in  this  replication  was  possibly 
weak.  Weakness  was  suggested  by  the  low  convergence  of  the 
superior  evaluations  with  the  other  criteria.  In  particular,  the 
correlations  between  rank,  awards,  and  subordinate  ratings  were 
as  expected:  moderately  high  (those  involving  subordinate 
ratings)  to  high  (between  rank  and  awards) .  In  contrast,  the 
correlations  of  these  criterion  measures  with  superior 
evaluations  were  uniformly  low  (even  if  statistically 
significant) .  Weakness  in  the  measurement  of  superior 
evaluations  could  have  reduced  the  sensitivity  of  this  test  in 
that  compared  to  subordinates,  superiors  can  possibly  more 
accurately  assess  the  effectiveness  of  organizational  problem 
solving.  Superiors,  for  example,  are  likely  better  positioned  to 
observe  the  effects  of  solutions  across  organizational  boundaries 
or  hierarchies,  to  include  their  effects  on  goals  set  by  higher 
echelons.  Finally,  this  replication  did  not  measure 
organizational  conditions  which  could  determine  the  influence 
leaders'  intellectual  abilities  on  organizational  problem  solving 
(Fiedler  &  Garcia,  1987) .  In  this  regard,  it  is  noteworthy  that 
with  the  exception  of  level  of  leadership,  the  problem-solving 
model  does  not  explicitly  treat  organizational  or  other  context 
variables . 

While  the  leadership  levels,  superior  measure,  and  neglect 
of  context  conditions  may  have  contributed  to  the  weaker  than 
expected  replication,  the  measures  of  the  predictors  also  deserve 
consideration.  As  mentioned  earlier,  most  of  the  predictor 
measures  were  new,  that  is,  developed  for  and  introduced  in  the 
original  effort.  Neither  the  original  research  nor  this 
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replication  included  already  validated  measures  for  comparison 
with  the  new  measures.  This  gap  was  a  mistake  in  that  the 
results  raise  questions  about  the  validity  of  the  new  measures. 
Results  generally  indicated: 

— Lack  of  convergent  and  divergent  validity.  Reasonable 
expectations  about  similarities  and  differences  in  the 
relationships  between  variables  or  variable  groups  were  not 
strongly  represented  in  the  patterns  of  correlations.  For 
example,  the  correlations  between  the  BDI  measures  within  the 
same  variable  domain  were  not  consistently  stronger  than  were  the 
correlations  between  variables  in  different  domains.  Similarly, 
the  correlations  between  the  scores  for  the  two  scenarios  in  a 
problem-solving  exercise  were  at  best  moderately  strong.  This 
lack  of  convergence  for  the  scenario-level  scores  was  also  shown 
in  the  factor  analysis  which  produced  scenario-specific  factors. 
Such  factors  raise  concerns  that  the  open-ended  responses  or  the 
scoring  processes  were  driven  by  the  scenarios  and  not  by  the 
intended  problem-solving  constructs. 

— Low  predictive  validity.  Few  of  the  BDI  scores  for 
cognitive  abilities  were  significantly  correlated  with  the  scores 
derived  from  the  open-ended  measures. 

--Contradictory  relationships.  Especially  for  the  BDI 
scales,  it  was  not  uncommon  that  significant  relations  showed  the 
unexpected  direction.  For  example,  preliminary  examination  of 
Table  A2  suggested  an  encouraging  trend.  That  is,  despite  the 
small  number  of  significant  correlations  (16  of  184),  practically 
all  (14  of  16)  were  obtained  for  ratings  given  either  by  or  to 
the  battalion  commander.  However,  further  inspection  showed  that 
most  (5  of  8)  of  the  significant  correlations  with  ratings  given 
to  battalion  commanders  were  in  the  unexpected  direction. 

It  is  important  to  recognize  that  the  support  from  this 
replication  is  limited  to  the  general  hypothesis  that  variables 
set  forth  in  the  model  account  for  variance  in  leadership 
effectiveness  as  indicated  by  career  achievement.  As  such,  the 
support  does  not  extend  to  the  model  as  fully  elaborated. 

Indeed,  this  replication  and  the  original  presentation  did  not 
directly  test  the  full  model.  Rather,  the  original  research 
partitioned  the  sample  and  used  regression  analyses  to  identify 
and  cross-validate  the  group  of  variables  which  best  accounted 
for  the  variance  in  the  original  criterion  measures.  This 
replication  followed  suit  and  sought  to  replicate  the  original 
effects  for  the  variable  group,  with  a  different  sample  and  with 
expanded  criteria.  Until  the  model  has  been  tested  with 
adequately  validated  predictors,  conclusions  about  the  promise  of 
the  model  would  be  premature. 

Given  the  findings,  applied  use  of  the  model  and  the 
measures  developed  to  test  it  would  also  be  premature.  As 
already  discussed,  the  validity  of  the  measures  is  open  to 
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question,  and  these  validity  questions  need  answers  prior  to  use 
of  the  measures  in  practice.  Measurement  reliability  also  needs 
consideration,  especially  the  reliability  of  scores  derived  from 
the  open-ended  problem-solving  exercises.  In  both  the  original 
presentation  and  in  this  replication,  the  reliability  of  the 
open-ended  scores  was  indexed  in  terms  of  the  consistency  with 
which  the  scorers  had  applied  the  rating  scales:  consistency 
among  scorers  in  the  values  assigned  to  a  particular  response 
relative  to  the  values  assigned  to  other  responses.  Such  an 
index  fails  to  indicate  the  similarity  of  the  values  which 
scorers  assign  to  any  response.  Similarity  between  scorers  in 
the  values  assigned  to  individual  responses  is  critical  for  the 
accurate  classification  of  individuals  and  for  decisions  based  on 
the  classification.  Prior  to  use  of  the  measures  in  practice, 
the  desired  level  of  similarity  needs  to  be  determined,  and  the 
extent  to  which  any  particular  measure  meets  the  desired  level 
needs  to  be  assessed. 

Inter-scorer  similarity  was  explored  for  responses  to  the 
scenarios  scored  for  problem  construction.  Each  of  three 
scorers  had  used  five-point  scales  to  rate  the  scenario  responses 
on  eight  dimensions  of  problem  construction.  With  a  five-point 
scale,  differences  in  the  values  of  any  rating  by  any  two  scorers 
could  have  ranged  from  zero  (no  disagreement)  to  four  (used 
opposite  ends  of  the  scale) .  To  assess  similarity,  each  of  300 
separate  ratings  was  classified  according  to  the  maximum 
difference  between  the  values  of  any  two  of  the  three  scorers  for 
the  rating.  No  disagreement  was  obtained  for  30  (10%)  of  the  300 
ratings.  For  another  136  ratings  (45%),  the  maximum  difference 
was  one  point  on  the  rating  scale.  For  100  ratings  (33%),  at 
least  two  of  the  three  scorers  differed  by  two  scale  points.  The 
maximum  difference  was  three  scale  points  for  another  28  ratings 
(9%),  with  only  one  rating  showing  a  maximum  difference  of  4 
scale  points.  While  this  index  does  not  directly  describe  the 
amount  of  actual  agreement  between  any  two  scorers,  one  can 
project  from  it  that  exact  agreement  ranged  from  32%  to  at  best 
45%  of  the  ratings;  it  also  shows  agreement  within  one  scale 
value  ranged  from  55%  to  at  best  66%. 

Research  on  the  problem-solving  model  is  continuing.  This 
research  should  include  a  focus  on  the  reliability  and  validity 
of  the  new  measures.  This  could  and  perhaps  should  involve  some 
combination  of  newer  data  with  the  older  data  sets.  Such  a 
combination  would  create  opportunities.  For  example,  neither  the 
original  sample  nor  the  replication  sample  was  sufficiently  large 
for  a  factor  analysis  of  the  responses  to  all  BDI  items.  A 
combination  of  samples  could  support  a  better  examination  of  the 
properties  of  the  whole  instrument.  The  continuing  research 
could  also  involve  some  re-analysis  of  the  replication  and 
original  data.  If,  for  example,  future  research  were  to  sharpen 
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the  coding  systems  for  open-ended  problem-solving  exercise,  the 
refined  systems  could  be  applied  to  earlier  data.  Such  a  re¬ 
analysis  could  be  useful  in  assessing  both  the  effects  of  the 
changes  and  in  determining  the  appropriateness  of  comparing 
findings  across  separate  investigations.  Such  a  re-analysis 
could  also  increase  confidence  in  conclusions  about  the 
scientific  and  practical  merits  of  the  problem-solving  model  of 
leadership . 
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Within  Rank  Correlations 
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Table  A-1. 

Correlations  between  BDI  Scales  and  Measures  Derived  from  Problem^-Solving  Exercises 
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